﻿When an HTTP client (generally a Web browser) requests a URL that points to a directory structure instead of an actual Web page within the directory, the Web server will generally serve a general page, which is often referred to as a main or "index" page.

index.html is the traditional filename for such a page, but most modern HTTP servers offer a configurable list of filenames that the server can use as an index. If a server is configured to support server-side scripting, the list will usually include entries allowing dynamic content to be used as the index page (e.g. index.php, index.shtml, default.asp) even though it may be more appropriate to still specify the html output (index.html.php or index.html.aspx), as this should not be taken for granted. An example is the popular open source web server Apache, where the list of filenames is controlled by the DirectoryIndex directive in the main server configuration file or in the configuration file for that directory. It is possible to make do without file extensions at all, and be neutral to content delivery methods, and set the server to automatically pick the best file through content negotiation.

If the server is unable to find a file with any of the names listed in its configuration, it may either return an error (generally 404 Not Found) or generate its own index page listing the files in the directory. It may also return a 403 Index Listing Forbidden. Usually this option is also configurable.

[edit] EntranceSince the index page is often the first page of a Web site that a user sees, it is sometimes used to offer a menu of language options for large Web sites that use geo targeting. It is also possible to avoid this step, for example by using content negotiation.

From Wikipedia, the free encyclopedia
Jump to: navigation, search 
Search engine indexing collects, parses, and stores data to facilitate fast and accurate information retrieval. Index design incorporates interdisciplinary concepts from linguistics, cognitive psychology, mathematics, informatics, physics, and computer science. An alternate name for the process in the context of search engines designed to find web pages on the Internet is Web indexing.

Popular engines focus on the full-text indexing of online, natural language documents.[1] Media types such as video and audio[2] and graphics[3][4] are also searchable.

Meta search engines reuse the indices of other services and do not store a local index, whereas cache-based search engines permanently store the index along with the corpus. Unlike full-text indices, partial-text services restrict the depth indexed to reduce index size. Larger services typically perform indexing at a predetermined time interval due to the required time and processing costs, while agent-based search engines index in real time.

Contents [hide]
1 Indexing 
1.1 Index Design Factors 
1.2 Index Data Structures 
1.3 Challenges in Parallelism 
1.4 Inverted indices 
1.5 Index Merging 
1.6 The Forward Index 
1.7 Compression 
2 Document Parsing 
2.1 Challenges in Natural Language Processing 
2.2 Tokenization 
2.3 Language Recognition 
2.4 Format Analysis 
2.5 Section Recognition 
2.6 Meta Tag Indexing 
3 Standardization 
4 See also 
5 Further reading 
6 References 
 
[edit] IndexingThe purpose of storing an index is to optimize speed and performance in finding relevant documents for a search query. Without an index, the search engine would scan every document in the corpus, which would require considerable time and computing power. For example, while an index of 10,000 documents can be queried within milliseconds, a sequential scan of every word in 10,000 large documents could take hours. The additional computer storage required to store the index, as well as the considerable increase in the time required for an update to take place, are traded off for the time saved during information retrieval.

[edit] Index Design FactorsMajor factors in designing a search engine's architecture include:

Merge factors  
How data enters the index, or how words or subject features are added to the index during text corpus traversal, and whether multiple indexers can work asynchronously. The indexer must first check whether it is updating old content or adding new content. Traversal typically correlates to the data collection policy. Search engine index merging is similar in concept to the SQL Merge command and other merge algorithms.[5] 
Storage techniques  
How to store the index data, that is, whether information should be data compressed or filtered. 
Index size  
How much computer storage is required to support the index. 
Lookup speed  
How quickly a word can be found in the inverted index. The speed of finding an entry in a data structure, compared with how quickly it can be updated or removed, is a central focus of computer science. 
Maintenance  
How the index is maintained over time.[6] 
Fault tolerance  
How important it is for the service to be reliable. Issues include dealing with index corruption, determining whether bad data can be treated in isolation, dealing with bad hardware, partitioning, and schemes such as hash-based or composite partitioning,[7] as well as replication. 
[edit] Index Data StructuresSearch engine architectures vary in the way indexing is performed and in methods of index storage to meet the various design factors. Types of indices include:

Suffix tree  
Figuratively structured like a tree, supports linear time lookup. Built by storing the suffixes of words. The suffix tree is a type of trie. Tries support extendable hashing, which is important for search engine indexing.[8] Used for searching for patterns in DNA sequences and clustering. A major drawback is that the storage of a word in the tree may require more storage than storing the word itself.[9] An alternate representation is a suffix array, which is considered to require less virtual memory and supports data compression such as the BWT algorithm. 
Inverted index  
Stores a list of occurrences of each atomic search criterion,[10] typically in the form of a hash table or binary tree.[11][12] 
Citation index  
Stores citations or hyperlinks between documents to support citation analysis, a subject of Bibliometrics. 
Ngram index  
Stores sequences of length of data to support other types of retrieval or text mining.[13] 
Document-term matrix  
Used in latent semantic analysis, stores the occurrences of words in documents in a two-dimensional sparse matrix. 
In computer science, an index can be:

an integer which identifies an array element 
a data structure that enables sublinear-time lookup 
[edit] Array element identifierWhen data objects are stored in an array, individual objects are selected by an index which is usually a non-negative scalar integer. Indices are also called subscripts.

There are three ways in which the elements of an array can be indexed:

0 (zero-based indexing) 
The first element of the array is indexed by subscript of 0. 
1 (one-based indexing) 
The first element of the array is indexed by subscript of 1. 
n (n-based indexing) 
The base index of an array can be freely chosen. Usually programming languages allowing n-based indexing also allow negative index values and other scalar data types like enumerations, or characters may be used as an array index. 
Arrays can have multiple dimensions, thus it is not uncommon to access an array using multiple indices. For example a two dimensional array A with three rows and four columns might provide access to the element at the 2nd row and 4th column by the expression: A[1,3] (in a row major language) and A[3,1] (in a column major language) in the case of a zero-based indexing system. Thus two indices are used for a two dimensional array, three for a three dimensional array, and n for an n dimensional array.

For details on programming language support of the various features see Comparison of programming languages (array).

[edit] Support for fast lookupSuppose a data store contains N data objects, and it is desired to retrieve one of them based on the value of one of the object's fields. A naive implementation would retrieve and examine each object until a match was found. A successful lookup would retrieve half the objects on average; an unsuccessful lookup all of them for each attempt. This means that the number of operations in the worst case is Ω(N) or linear time. Since data stores commonly contain millions of objects and since lookup is a common operation, it is often desirable to improve on this performance.

An index is any data structure which improves the performance of lookup. There are many different data structures used for this purpose, and in fact a substantial proportion of the field of Computer Science is devoted to the design and analysis of index data structures. There are complex design trade-offs involving lookup performance, index size, and index update performance. Many index designs exhibit logarithmic (O(log(N)) lookup performance and in some applications it is possible to achieve flat (O(1)) performance.

All database software includes indexing technology in the interests of improving performance. See Index (database).

One specific and very common application is in the domain of information retrieval, where the application of a full-text index enables rapid identification of documents based on their textual content.

A database index is a data structure that improves the speed of data retrieval operations on a database table at the cost of slower writes and increased storage space. Indexes can be created using one or more columns of a database table, providing the basis for both rapid random lookups and efficient access of ordered records. The disk space required to store the index is typically less than that required by the table (since indices usually contain only the key-fields according to which the table is to be arranged, and exclude all the other details in the table), yielding the possibility to store indices in memory for a table whose data is too large to store in memory.

In a relational database, an index is a copy of one part of a table. Some databases extend the power of indexing by allowing indices to be created on functions or expressions. For example, an index could be created on upper(last_name), which would only store the upper case versions of the last_name field in the index. Another option sometimes supported is the use of "filtered" indices , where index entries are created only for those records that satisfy some conditional expression. A further aspect of flexibility is to permit indexing on user-defined functions, as well as expressions formed from an assortment of built-in functions.

Indices may be defined as unique or non-unique. A unique index acts as a constraint on the table by preventing duplicate entries in the index and thus the backing table.

Contents [hide]
1 Index architecture 
1.1 Non-clustered 
1.2 Clustered 
2 Column order 
3 Applications and limitations 
4 Types 
4.1 Bitmap index 
4.2 Dense index 
4.3 Sparse index 
4.4 Reverse index 
5 Index implementations 
6 Covering Index 
7 Standardization 
8 See also 
9 References 
10 External links 
 
[edit] Index architectureIndex architectures can be classified as clustered or nonclustered.

[edit] Non-clusteredThe data is present in random order, but the logical ordering is specified by the index. The data rows may be randomly spread throughout the table. The non-clustered index tree contains the index keys in sorted order, with the leaf level of the index containing the pointer to the page and the row number in the data page. In non-clustered index:

The physical order of the rows is not the same as the index order. 
Typically created on column used in JOIN, WHERE, and ORDER BY clauses. 
Good for tables whose values may be modified frequently. 
Microsoft SQL Server creates non-clustered indices by default when CREATE INDEX command is given. There can be more than one non-clustered index on a database table. There can be as many as 249 nonclustered indexes per table. It also creates a clustered index on a primary key by default.[1]

[edit] ClusteredClustering alters the data block into a certain distinct order to match the index, resulting in the row data being stored in order. Therefore, only one clustered index can be created on a given database table. Clustered indices can greatly increase overall speed of retrieval, but usually only where the data is accessed sequentially in the same or reverse order of the clustered index, or when a range of items is selected.

Since the physical records are in this sort order on disk, the next row item in the sequence is immediately before or after the last one, and so fewer data block reads are required. The primary feature of a clustered index is therefore the ordering of the physical data rows in accordance with the index blocks that point to them. Some databases separate the data and index blocks into separate files, others put two completely different data blocks within the same physical file(s). Create an object where the physical order of rows is same as the index order of the rows and the bottom(leaf) level of clustered index contains the actual data rows.

They are known as "index organized tables" under Oracle database.

[edit] Column orderThe order in which columns are listed in the index definition is important. It is possible to retrieve a set of row identifiers using only the first indexed column. However, it is not possible or efficient (on most databases) to retrieve the set of row identifiers using only the second or greater indexed column.

For example, imagine a phone book that is organized by city first, then by last name, and then by first name. If you are given the city, you can easily extract the list of all phone numbers for that city. However, in this phone book it would be very tedious to find all the phone numbers for a given last name. You would have to look within each city's section for the entries with that last name. Some databases can do this, others just won’t use the index.

[edit] Applications and limitationsIndices are useful for many applications but come with some limitations. Consider the following SQL statement: SELECT first_name FROM people WHERE last_name = 'Smith';. To process this statement without an index the database software must look at the last_name column on every row in the table (this is known as a full table scan). With an index the database simply follows the B-tree data structure until the Smith entry has been found; this is much less computationally expensive than a full table scan.

Consider this SQL statement: SELECT email_address FROM customers WHERE email_address LIKE '%@yahoo.com';. This query would yield an email address for every customer whose email address ends with "@yahoo.com", but even if the email_address column has been indexed the database still must perform a full table scan. This is because the index is built with the assumption that words go from left to right. With a wildcard at the beginning of the search-term, the database software is unable to use the underlying b-tree data structure (in other words, the WHERE-clause is not sargable). This problem can be solved through the addition of another index created on reverse(email_address) and a SQL query like this: SELECT email_address FROM customers WHERE reverse(email_address) LIKE reverse('%@yahoo.com');. This puts the wild-card at the right-most part of the query (now moc.oohay@%) which the index on reverse(email_address) can satisfy.

[edit] Types[edit] Bitmap indexMain article: Bitmap index
A bitmap index is a special kind of index that stores the bulk of its data as bit arrays (bitmaps) and answers most queries by performing bitwise logical operations on these bitmaps. The most commonly used index, such as B+trees, are most efficient if the values it indexes do not repeat or repeat a smaller number of times. In contrast, the bitmap index is designed for cases where the values of a variable repeat very frequently. For example, the gender field in a customer database usually contains two distinct values: male or female. For such variables, the bitmap index can have a significant performance advantage over the commonly used trees.

[edit] Dense indexA dense index in databases is a file with pairs of keys and pointers for every record in the data file. Every key in this file is associated with a particular pointer to a record in the sorted data file. In clustered indices with duplicate keys, the dense index points to the first record with that key.[2]

[edit] Sparse indexA sparse index in databases is a file with pairs of keys and pointers for every block in the data file. Every key in this file is associated with a particular pointer to the block in the sorted data file. In clustered indices with duplicate keys, the sparse index points to the lowest search key in each block. primary key is a sparse index.

[edit] Reverse indexMain article: Reverse index
A reverse key index reverses the key value before entering it in the index. E.g., the value 24538 becomes 83542 in the index. Reversing the key value is particularly useful for indexing data such as sequence numbers, where new key values monotonically increase.

[edit] Index implementationsIndices can be implemented using a variety of data structures. Popular indices include balanced trees, B+ trees and hashes.[3]

In Microsoft SQL Server, the leaf node of the clustered index corresponds to the actual data, not simply a pointer to data that resides elsewhere, as is the case with a non-clustered index.[4] Each relation can have a single clustered index and many unclustered indices.[5]

[edit] Covering IndexIn most cases, an index is used to quickly locate the data record(s) from which the required data is read. In other words, the index is only used to locate data records in the table and not to return data.

A covering index is a special case where the index itself contains the required data field(s) and can return the data.

Consider the following table (other fields omitted):

ID Name Other Fields 
12 Plug ... 
13 Lamp ... 
14 Fuse ... 

To find the Name for ID 13, an index on (ID) will be useful, but the record must still be read to get the Name. However, an index on (ID, Name) contains the required data field and eliminates the need to look up the record.

A covering index can dramatically speed up data retrieval but may itself be large due to the additional keys, which slow down data insertion & update. To reduce such index size, some systems allow non-key fields to be included in the index. Non-key fields are not themselves part of the index ordering but only included at the leaf level, allowing for a covering index with less overall index size.
